DOCUMENT RESUME 



ED 376 790 



IR 016 841 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Burnham, Byron R. 

A Systemic View of Distance Education Evaluation: 
Finding Our Bearings* 
11 Aug 94 

13p.; Paper presented at the Annual Conference on 
Distance Teaching and Learning (10th, Madison, WI, 
August 1994) ♦ 

Speeches/Conference Papers (150) — Viewpoints 
(Opinion/Position Papers, Essays, etc.) (120) 

MF01/PC01 Plus Postage. 

Conventional Instruction; *Di stance Education; 
Educational Research; '''Educational Technology; 
Elementary Secondary Education; '''Evaluation Methods; 
Higher Education; Interaction; *Models; 
Nontradit ional Education; Program Evaluation; 
'''Systems Approach; ^Teaching Methods 



ABSTRACT 

For a number of years it has been suggested that 
distance education is a natural extension of existing teaching 
methods that includes learners who are separated. from the instructor 
by time or distance. This paper takes an alternative look at 
distance-education research and evaluation and proposes an evaluation 
model built on that view. That distance education is essentially 
different from traditional education is made apparent by examining 
both methods and techniques. Methods generate techniques, and both 
must be distinguished from the devices (technology hardware) that are 
used in education. In traditional instruction the flow is from method 
to technique to device; in distance education, the device can affect 
method and technique. An evaluation framework is diagrammed that 
depicts the relations among device, method, technique, learner, 
content, and instructor. The interaction among these elements cannot 
be ignored during the development of distance education in the 
future. Three figures and two tables illustrate the framework. 
(Contains 7 references.) (SLD) 
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Distance Education is a growing and developing field in a number of regards. 
First, it is growing in the number of people participating. Second, it is growing in the 
number of organizations which sponsor distance education activities. Third, it is 
growing in the kinds of organizations which sponsor distance education. Fouth, it is 
developing in sophistication of programs. And fifth, it is developing in the 
sophistication of methods. 

To accompany these areas of growth and development a correlative area must be 
developed. For a number of years the idea that distance education is a natural 
extension of existing classes, meetings, or seminars which include learners who are 
separated from the instructor in either time or distance, or both have guided our 
activities in distance education research and evaluation. This paper suggests an 
alternative look at distance education research and evaluation and proposes an 
evaluation model built upon that alternative view. 

That distance education should be recognized essentially as a different kind of 
endeavor from traditional face to face education has been recognized by those working 
and writing in the field of distance education and those working in closely related 
fields. For example, Garrison (1989) offers three criteria as minimum characteristics 
of a distance education setting. He notes that in distance education the majority of 
educational communication occurs noncontiguously, there will be two-way 
communication between teacher and student taking place, and communication will be 
technologically mediated. 

These characteristics are provided by a distance educator and reflect the field 
view of distance education. Closely related to this definition is the American 
Association for the Advancement of Sciences define .on of technology (1989). "In the 
broadest sense, technology extends our ability to change the world: to cut, shape, or 
put together; to move things from one place to another; to reach further with our 
hands, voices, and senses* (p.39). While this definition is not about distance education 
per se, it does describe what happens in a distance education setting. 
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While various authors writing in the field of distance education, and those 
writing in closely allied fields, find common or related perspectives there remains a 
problem with those who look at distance education with only technology in mind. This 
fact is lamented by Hofineister, Carnine, and Clark (1993) when they say "Many of the 
present conceptual structures surrounding technology, media, and materials are 
hardware-focused and detract from more potent variables, such as the content and 
structure of the curriculum, the supporting pedagogy, and the interaction between the 
teacher and the techrology, media, and materials.... We diminish the values of and the 
focus on this dissemination role when hardware is promoted with questionable claims 
implying unique and powerful contributions to the learning process* (p. 2). 

These problems can be greatly reduced if the phenomenon of distance education 
is conceptually viewed in a holistic manner. It is the purpose of this paper to present 
one such view. 

An updated framework 

In the early 1960s Coolie Verner proposed a conceptual scheme for classifying 
various components of the educational transaction. He did this because he felt that 
"the generally recognized confusion that characterized adult education stems from the 
absence of any conceptual scheme or basic theoretical structure" (p. iii), a condition 
which exists in distance education today. Verner's framework depends upon the 
precise use of two popularly confused terms: methods and techniques. To Verner they 
meant vastly different things and those differences can and should inform our practice 
of distance education. 

Verner defined method as "the relationship established by the institution with a 
potential body of participants for the purpose of systematically diffusing knowledge 
among a prescribed by not necessarily fully identified public" (p. 9). This idea of 
method is limited to organizational and sociological concerns. It does not encompass 
the psychological construction of learning. While much effort has been directed toward 
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learning in our studies of DE, little has been aimed at the organization and social 
relationships in DE. 

Technique was defined as "the relationship established by the institutional agent 
to facilitate learning among a particular and precisely defined body of participants in a 
specific situation" (p. 9). Verner's notion was that methods refer to the way in which 
organizations relate to groups of people and techniques are the ways people relate to 
information for the purpose of learning. ' 

Methods, according to Verner, provide us with a set of techniques appropriate to 
given situations. Methods can be individual in nature, such as apprenticeships, 
correspondence study or internships. Group methods can be classified into natural 
groups (such as a family or autonomous groups) or social groups (such as formal 
associations or work groups). Small groups, larger groups, community groups are all 
possible classifications. 

Verner understood the place of technology hardware, or as he labeled them, 
"devices." This notion of devices is particularly informative and places a new light on 
what we term technology. Verner stated "In producing educational activities for adults 
numerous mechanical instruments or environmental factors may be employed to 
augment the processes employed. These are frequently identified by the field as 
methods or techniques when in reality they should be designated more precisely as 
devices * since they enhance the effectiveness and utility of techniques but cannot 
themselves function independently as techniques for the acquisition of knowledge" (p. 



Verner also understood any method of education was the function of the 
"continuing relationship for systematic learning that is established by the institution 
with those in the public whom it seeks to educate..." (p. 13). Then, according to this 
description of method, individual and institutional needs combine to determine the 
method. Verner implies in his writings that some techniques are method bound. 
Certainly, an instructional designer would not employ a group discussion technique in 
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a method in which learners exist in isolation (i.e. an apprenticeship). Once methods 
and techniques have been selected, then devices can be chosen, because devices, in and 
of themselves, do not teach. 

The flow of the traditional instructional design process is from method to 
technique to device. Verner in the early 1960's could not anticipate the advent of the 
smart devices which exist today. Burnham and Seamons (1987) proposed "the idea 
that devices, especially electronic devices and systems, can affect methods or even 
create methods unanticipated by Verner.. (p. 10). These new methods have not 
received much attention from the researchers in distance education. We have been so 
caught-up with demonstrating that there are no statistical differences between 
distance education and face to face education that we seem to have forgetten the notion 
that we are dealing with another kind of educational setting or method. 

To illustrate the differences, consider for a moment the traditional classroom 
method of education where students are assigned to a class or register for one. While it 
is true that the student-makeup of today's university classes is diverse, it is also true 
that the student-makeup of distance education classes is of even greater diversity. 
Consider for a moment the kinds of students that may be in th* same distance 
education class: displaced homemakers, unemployed space industry workers, and 
inmates from the state prison. How do these people interact with one another? * 
Another and more fundamental question is: What constitutes a classroom or learning 
group? Is it the total of the people at the various sites or should we consider each 
small receiving site as a classroom? It is important to realize that the classroom 
method is changed by the new devices we encounter in distance education. We are 
mistakenly attributing the affects in learning to the devices when we might be better 
advised to consider the effects of methods and techniques which are enabled or 
disabled by devises. 

This is precisely the point made by in the Blueprint for Action (Hofmiester, et 
al., 1993). The advantage of using the labels with specific references to particular 
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areas of education means that differentiation can be made among technology (the 
devices), the administration of distance education (the methods), and instructional 
design matters or processes for facilitating learning (the techniques). Such 
differentiation will help avoid the problems noted by Burnham and Seamons (1987) as 
they described a metaphor of distance education. "The current landscape of Electronic 
Distance Education (EDE) is broken and confused by boulders of hardware, rivers of 
processes, and mountains of exhortations.... Because of technological changes and 
improvements, the future seems encumbered with even more conceptual confusion and 
technological preoccupations than is presently the case" (p.8). The comments of the 
AAAS 2061 project staff would indicate that the statement was prophetic to some 
degree. 



A distance education system can be simply represented by Figure 1 wherein the 




Figure L An Electronic Distance Education System 
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relationship of devices, methods, techniques, learners, content and instructors are 
portrayed in relationship to one another. 
An evaluation framework 

This conceptual framework can be used to develop a research or an evaluation 
framework for distance education. Such a framework allows researchers and 
evaluators to understand what area of distance education they are studying thus 
avoiding the problem of thinking about distance education as some amorphous mass 
which is treated similarly no matter what is being studied. The framework also can 
help direct investigators into methodological considerations as well as measurement 
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Figure 2. An evaluation/research model for distance education 
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concerns. The framework presented in Figure 2 is built upon the considerations 
described above. 

This framework provides help in understanding how various parts of distance 
education may or may not relate to one another. For example, issues of satisfaction 
with the system, administrative support, personal learning styles, behavioral changes 
and the like can be studied from the learners' perspectives or within the contexts of 
methods, techniques, and devices. Learners can provide perspectives on the learning 
environment (both methods and techniques) that can guide instructor behavior. 

The perspective of the instructors about various issues can provide information 
about preparation time, perceived effectiveness, satisfaction, use and perceptions of 
instructional techniques, techniques and administrative support. Methodological 
implications (in the research and evaluation sense) include measurement, 
observational, and interview approaches. Linking these two elements of distance 
education (instructor and learner perspectives) are the characteristics of devices used. 

A number of studies have been conducted which are diverse in nature, but are 
grounded in the frameworks presented above. To demonstrate how the framework can 
help integrate the various findings, a brief description of different studies are produced 
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||||: : .Primary Method 


Findings ? 


Course evaluation of DE 
classes/learner perspective 


Survey/Quantitative 


Distance Education instructors 
are rated similarly to face to 
face instructors. 


Evaluation of audio course/ 
learner perspective 


Site observations/Qualitative 


Technology interfered with 
interactions. Parallel learning 
takes place. 


The devises used in DE 
(Development of a technology 
rating instrument/expert 
opinion) 


Telephone survey 
(Quantitative) 


Communication, ease of use, 
realism, independence, and 
speed are ranked 
characteristics of DE devices 
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Study of participant Survey/Quantitative 

motivations/learner 

perspective 



DE student motivations are 
similar to face to face student 
motivations. Motivational 
orientations do not account for 
satisfaction differences. 



A study of liberal arts courses 
at a distance/instructor 
perspective 



Interviews, Content Analysis/ 
Qualitative 



Learning time can be 
compressed with use of extended 
syllabi. Instructors have high 
regard for DE students. 



Study of instructional 
motivation/learner perspective 



Survey/Quantitative 



DE students are motivated by 
similar instructional design 
elements as face to face 
students. 



in the table below which shows the focus and perspectives of the study, the methods 
employed and one or two findings from the investigations. 

The studies reviewed here provide a variety of findings which all have 
implications for distance education. The studies all have v rying focuses and employ 
various research methods. A holistic perspective is provided by employing the 
conceptual analysis described at the beginning of the paper. It is evident which studies 
apply to devices. However, it is more difficult to differentiate method and techniques 
unless one knows the scope of the study. All of the studies here would be classified as 
dealing with distance education methods because they deal with wider concepts and 
issues beyond the learning techniques employed. Techniques investigation would be 
limited within a method (distance education, class, or individual methods) and look at 
the way people learn under specific learning techiques. 

Parallel learning 

Of the findings listed in the table above, one of the most interesting is the notion 
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of parallel learning. This finding has implications for instructional design 
(techniques), distance education administration (methods), and technology (devices). 
To better illustrate these implications a brief description of parallel learning is offered 
here. 

Parallel learning describes that activity in which adult distant education 
students engage whenever we as instructors become boring or irrelevant (I know that 
happen only rarely, but it does happen). DE provides a setting where adults can and 
do talk to one another about what is being discussed without being rude to the teacher. 
And during these discussions they process information that is relevant to the topic at 
hand. 

We have found from observing DE at remote sites that they do not sit passively 
awaiting instruction from all knowing experts. To the contrary we found adults 
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Figure 3. Instructional Stream 
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actually guessing what the expert was going to say next, or debating what had been 
said. They interacted with the instructor but they interacted more frequently and 
longer with fellow students at the local site. This interaction produced insights which 
paticipants indicated would change their practices. 

Figure 3 illustrates what might be called the instructional stream which is 
largely under the direction of the instructor. However, there are events which 
encourage DE students to engage in the behavior of parrallel learning. There are 
other episodes which cannot be classified as parallel learning, which are actually non- 
parallel in nature. 

The trigger events may be a distruption in the signal to the remote site, an 
especially boring lecture, unusally controversial statements by the instructor, 
irrelevant statements, etc. Less well documented are those events which return the 
learners to the instructional stream. 



Implications 

The implications for the instructional designer are to help teachers aviod 
unintended trigger events and to maximize their effect when appropriate (techniques). 
For the technology system manager implications have to do with the smooth running of 
a system (devices). And for the distance education adminstrator implications are for 
the socialization of learners to a distance education system (methods). 

I am concerned that instructional development will in the future be concerned 
with message design, graphics, public speaking, and broadcast mannerisms and will 
neglect the part of information dismessination that turns it into education: 
interaction. When you think about it, satellite distance education is not much different 
from watching television. The only difference is that I'm much more comfortable 
watching TV in my home than a conference room. 

Methods, techniques, and devices are all elements that interact much more that 
previous researchers and theorticians have suspected. We need to study that 
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r interaction so as to better understand the dymanic enterprize called distance 
education. 
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